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ABSTRACT 



This report describes activities and achievements of a 
project that examined state policies linking new accountability demands with 
improvements in college teaching and learning. Major activities included: (1) 

meetings and national forums to explore state and public concerns about the 
quality and results of undergraduate education; (2) two commissioned papers 
and 10 case studies on state policy initiatives supporting improvements in 
teaching and learning through the development of performance indicators; and 
(3) wide dissemination of the project’s findings and continued involvement of 
the Education Commission of the States (ECS) in the development of 
performance indicators for higher education. The state-level case studies 
provide in-depth examinations of formulation, planning, implementation, and 
reviews of policies and practices related to the assessment of education 
effectiveness in Colorado, Florida, Illinois, Kentucky, New York-SUNY (State 
University of New York), South Carolina, Tennessee, Texas, Virginia, and 
Wisconsin. After a project overview, individual sections of the report 
describe the project's purposes, background and origins, activities, 
evaluation, and results. Attachments include a brochure about ECS, the case 
study letter and list of contacts, the interview format, and the seminar 
letter and roster of participants. Also attached is the project's major 
product, a sourcebook on state-level performance indicators titled "Charting 
Higher Education Accountability," (Sandra S. Ruppert, Ed.) which contains the 
full case studies plus two essays, "Effectiveness in Undergraduate Education: 
An Analysis of State Quality Indicators" (Richard C. Richardson, Jr.) and 
"Developing Statewide Performance Indicators for Higher Education: Policy 
Themes and Variations" (Peter T. Ewell) . (DB) 
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PROJECT SUMMARY 



The New Accountability: 

Policies to Improve Teaching and Learning 



College-level "student assessment" and "institutional accountability" for undergraduate 
education outcomes developed along parallel tracks throughout the 1980s. A convergence of 
interest in student learning and institutional performance became more apparent after 1990 
and was captured by the phrase the "new accountability." States, in particular, became more 
interested in the "returns on their investment" in undergraduate education raising questions 
such as how these could be defined more clearly, what policies supported improvement in 
college teaching and learning, and how progress (and comparative results) could be monitored 
across institutions. This project achieved significant results in several areas, including: 

• meetings and national forums to explore the stale and public concerns about the 
quality and results of undergraduate education, 

• commissioned papers and case studies on state policy initiatives to support 
improvements in teaching and learning through the development of performance 
indicators, and 

• wide dissemination of the project’s findings combined with continued ECS 
involvement in the development of performance indicators for higher education. 



ECS Contact: 
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EXECUTIVE SUMMARY 



The New Accountability: 

Policies to Improve Teaching and Learning 
Education Commission of the States 
707 17th Street, Suite 2700 
Denver, CO 80202-3427 

ECS Contact: Charles S. Lentil, 303-299-3667 
Project Director: Sandra Ruppert (8/94 Departure from ECS) 



PROJECT OVERVIEW 

This project developed from prior efforts by the Education Commission of the States (ECS) to analyze 
and improve state policies for student assessment and public accountability, and from the 
organization’s continuing involvement with state political and higher education leadership. Through 
the New Accountability: Policies to Improve Teaching and Learning, ECS sought to provide 
public policymakers with new approaches to address the dual purposes of (I) determining if 
students collectively are developing the skills, knowledge and underlying values that contribute to 
meeting the public goals of higher education; and (2) triggering the institutional improvements in 
teaching and learning necessary to meet these goals more effectively. 

Project activities and results included: meetings and national forums to explore the state and public 
concerns about the quality of undergraduate education; commissioned papers and 10 case studies on 
state policy initiatives to support improvements in teaching and learning through the development of 
performance indicators; and wide dissemination of the project’s key report. Charting Higher Education 
Accountability: A Sourcebook on State-Level Performance Indicators, combined with continued ECS 
involvement in the development of performance indicators for higher education. 

PURPOSES 

During the 1980s, the emphasis in state assessment policies was on institutional improvement, with 
accountability viewed more as a "by-product" of these improvements. In ECS’s 50-state survey 
conducted in 1989, for example, the majority of 40 states with assessment policies in place indicated 
that institutional self-improvement, not accountability, was the primary purpose for these policies. By 
1990, however, states were becoming more insistent in using both assessment and accountability as a 
means to enhance the educational effectiveness of public colleges and universities. 

The phrase "new accountability" described the gradual shift in public policy purposes to link 
institutional accountability more closely to demonstrated improvements in teaching and learning 
incorporating student learning outcomes into a broader set of concerns about higher education’s 
quality, effectiveness and productivity. Although the immediate causes varied from state to state, 
greater accountability was being driven as well by decreased state resources for higher education, 
rising costs, and growing demands for access. 

Many states joined in the effort to develop common measures for assessing and reporting the 
performance of colleges and universities. By mid- 1994, approximately one-third of states had some 
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form of "performance indicators" in place, with the vast majority developed and implemented in the 
previous three years. 

Within this changing context, the project had three related purposes: (1) to analyze and help states 
think through the design and implementation of policies that relate student assessment to institutional 
accountability; (2) to relate assessment and accountability policies to other areas and purposes of state 
higher education policy, such as academic planning, finance and governance; and (3) to host forums 
and produce printed materials useful to all states in dealing with the linking of assessment and 
accountability. 

BACKGROUND AND ORIGINS AT ECS 

In 1985, the ECS publication. Transforming the State Role in Undergraduate Education , recommended 
that higher education report on the extent to which it meets state goals for undergraduate education. 
Subsequent national surveys of state assessment initiatives by ECS led to the publication. Assessing 
College Outcomes - What State Leaders Need to Know , which provided guidance about using 
assessment policy to benefit both students and states. 

The "new accountability" project built directly on this past work. Since the primary constituents of 
ECS include governors, legislative leaders and their senior policy aides, state higher education 
executive officers, leaders of campuses and governing bodies, and others, ECS brought to tliis project 
the experience and capacity to bring relevant stakeholders together to discuss common problems across 
the states, draw on existing research and practices in the area of assessment and accountability, and 
relate assessment and accountability to K-12 education concerns as well as the broader ECS agenda 
encompassing system change. 

PROJECT DESCRIPTION 

The project began with a series of dialogues to (1) examine the issues of institutional quality and 
effectiveness and discuss practices related to good undergraduate education, (2) examine the evidence 
or information that would indicate if conditions and practices related to high-quality undergraduate 
education exist on a campus, and (3) analyze the input, process or outcome indicators that also would 
be helpful for institutions to improve student learning. 

A seven-member planning group was formed to assist in guiding and evaluating the project. Two 
papers were commissioned to examine and help states develop conceptual frameworks for the 
development of performance indicators, and two regional meetings were held to debate and refine 
these ideas. ECS commissioned an additional working document to provide a "common language" for 
discussing performance indicators. "Pointing the Way: Indicators as Policy Tools in Higher 
Education," prepared by Peter Ewell and Dennis Jones of NCHEMS, describes indicator systems and 
how they work, and provides a set of policy-level questions for reviewing the soundness of any 
particular indicator proposal. 

The project also included in-depth examination of 10 state-level case studies (Colorado, Florida, 
Illinois, Kentucky, New York-SUNY, South Carolina, Tennessee, Texas, Virginia and Wisconsin) 
developed to analyze the prevailing state practices and experiences. The case studies, based on the 
interviews and materials provided by the state agencies, provide history of the formulation, planning, 
implementation, and review of policies and practices related to the assessment of education 
effectiveness. ECS also hosted a seminar on "Indicators of Quality and Accountability for Higher 
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Education Systems" for representatives from the 10 case-study states in conjunction witli the ECS 
National Forum and Annual Meeting. 

The project concluded with the publication of Charting Higher Education Accountability: A 
Sourcebook on Stale-Level Performance Indicators , which analyzes how performance measures or 
indicators were developed, and for what purposes the results are being used. 

EVALUATION/PROJECT RESULTS 

This project has produced visible and demonstrable results from the activities outlined above. The 
advisory group meetings and regional forums, which were central to shaping the course of this project, 
served as mechanisms for interim evaluation and course correction. Although, no formal evaluation of 
the separate activities was undertaken, periodic feedback served an important role in formative 
assessment and evaluation. In addition, stale higher education agencies were repeatedly contacted to 
evaluate our interpretation of developments in their state and provide valuable feedback. 

An important indicator of this project’s results is the widespread use of the materials produced. 
Participants "took home what they learned" in order to shape and redirect some of their own actions. 
Copies of Charting Higher Education Accountability have been sent to all state higher education 
agencies and other interested parties; numerous statements have been received about its value and 
usefulness along with numerous requests for additional copies. Several requests also have been 
received from faculty members in graduate higher education programs for multiple copies to use in 
teaching courses on state policy. Findings from this project also have been presented at national and 
state-level meetings. Additional presentations and short articles in widely-distributed ECS publications 
(e.g., State Leader) are planned for the near future. 

Finally, the positive evaluation and results of this project convinced ECS to undertake an additional 
summary report. Scheduled for publication in November, Charting Higher Education Accountability: 
The Roots and Realities of State-Level Performance Indicators will be aimed at governors and 
legislators, legislative and executive assistants, state higher education leadership and others in 
policymaking positions, and will summarize and set forth the principal findings. 

SUMMARY AND CONCLUSIONS 

This project has been successful in meeting several important objectives and contributing to the 
continuing development of accountability policies and practices in many settings. Specifically, we 
explored and analyzed the complexities of student assessment and institutional accountability policies 
at the state level; we worked with states to relate assessment and accountability to other areas of state 
policy, including academic planning, finance and governance; and we organized numerous meetings, 
made presentations and produced materials that extended the reach of tliis project to many states, 
organizations and institutions. 

Interest in the issues of the quality and performance of colleges and universities is continuing to grow. 
ECS will continue to be an active partner with states in articulating public expectations for higher 
education more clearly and developing indicator systems that monitor progress in acliieving these 
results. 
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EDUCATION COMMISSION OF THE STATES 



The New Accountability: 

Policies to Improve Teaching and Learning 

A Final Report Submitted to the 
Fund for the Improvement of Postsecondary Education 

PROJECT OVERVIEW 

This project developed from ECS’s prior work with respect to state policies for student 
assessment and public accountability, and from the organization’s direct involvement with 
state political and higher education leadership. Drawing from previous work, including the 
ECS publications. Transforming the State Role in Undergraduate Education and Assessing 
College Outcomes - What State Leaders Need to Know , this project examined the convergence 
of interest in student assessment and institutional accountability through regional and national 
meetings, analytic papers, and case studies of 10 states’ development of higher education 
performance indicators. 

Over the course of this project, the focus was both broadened and sharpened — 
broadened in the recognition that neither assessment nor accountability can be viewed in 
isolation from the other or from the changing and diverse social and economic contexts of 
higher education; and sharpened to focus on the central importance of how we define and 
report on quality and performance in higher education as potentially powerful forces in 
shaping state policy and, in turn, higher education. 

The final project report is the most concrete manifestation of these efforts. Charting 
Higher Education Accountability: A Sourcebook on State-Level Performance Indicators 
includes two analytic papers on the development and use of performance indicators — 
"Effectiveness in Undergraduate Education: An Analysis of State Quality Indicators," by 
Richard C. Richardson, Jr. and "Developing Statewide Performance Indicators for Higher 
Education: Policy Themes and Variations," by Peter T. Ewell. These papers provided 
direction both for this project and for many states developing their own form of state "report 
card" or other monitoring procedure. 
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In addition, the sourcebook contains the state case studies tracking the development of 
performance indicators within 10 diverse settings. It discloses how these new accountability 
policies connect with other areas of state higher education policy and how each state 
evaluates the results of these efforts. Finally, this major project publication includes an 
interpretive essay, "Pointing the Way: Indicators as Policy Tools in Higher Education," by 
Peter T. Ewell and Dennis P. Jones, that will be helpful to states and other organizations in 
addressing the needs for more flexible and effective relationships between state governments 
and our colleges and universities. ECS will, in the near future, summarize and disseminate 
the key findings from this project to a wider audience through a "policy guide" to be 
distributed to governors, state legislators, institutional leaders and other interested parties, and 
through articles in ECS communication vehicles such as the State Leader. 

PURPOSES 

As initially conceived, the New Accountability: Policies to Improve Teaching and 
Learning was undertaken as a project of the Education Commission of the States (ECS) to 

provide public policymakers with new approaches to address the dual purposes of (1) 
determining if students collectively are developing the skills, knowledge and underlying 
values that contribute to meeting the public goals of higher education; and (2) triggering 
the institutional improvements in teaching and learning necessary to meet these goals 
more effectively. 

When the preliminary proposal was submitted to FIPSE in late 1990, it was clear that 
states were becoming more insistent in using assessment as a tool to gauge the educational 
effectiveness of public colleges and universities and as a means to improve teaching and 
learning. At that time, the emphasis in state assessment policies was on institutional 
improvement, with accountability viewed more as a "by-product." According to ECS’s 50- 
state survey conducted in 1989, for example, the majority of the 40 states with assessment 
policies in place indicated that institutional self-improvement, not accountability, was the 
primary purpose for assessment. 

The phrase "new accountability" (coined by one of the key consultants on the project, 
Peter Ewell), described a gradual shift in public policy purposes to link institutional 
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accountability more closely to demonstrated improvements in teaching and learning. Concern 
was being expressed by Ewell and other thoughtful observers that if ways were not found to 
address issues related to both accountability and improvement, "the movement to assess 
student learning, left to progress somewhat independently at state and institutional levels, runs 
the risk of developing at cross purposes with the need for accountability." 

While campus-based assessment led to some demonstrated improvements in teaching 
and learning at the level of the classroom or department, the results of assessment were 
neither as far-reaching nor as deep as policymakers had hoped. Institutional assessment had 
only limited success in impacting the way colleges and universities set priorities for their 
undergraduate teaching mission. Moreover, answers were yet to be provided to the broader 
state and public questions about higher education’s escalating costs and the "return on 
investment." Additional questions about "what college graduates know and are able to do" 
were not directly addressed — at least not on a collective, statewide basis — through the 
focus on institution-based assessment. 

Consequently, public officials increasingly incorporated their concerns for student 
learning outcomes into a broader set of issues about higher education’s accountability for 
quality, effectiveness and productivity. This move toward greater accountability was being 
driven as well by decreased state resources for higher education, rising costs and growing 
demands for access. Although the immediate causes varied from state to state, policymakers 
began to look seriously at using state-level accountability measures to guide state planning 
and budgetary decisions and to monitor the "public investment" in higher education. 

Over the course of this project, and in consultation with FIPSE staff, the purposes of 
this project were modified to reflect and contribute directly to this evolving debate over 
assessment and accountability. This resulted in three categories of project goals and 
activities: 

1. To analyze and help states think through the design and implementation of policies 
that relate student assessment to institutional accountability; 

2. To relate assessment and accountability policies to other areas and purposes of state 
higher education policy, such as academic planning, finance and governance; and 
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To host forums and produce printed materials useful to all states in dealing with the 
linking of assessment and accountability. 

During the course of this project, many states joined in the effort to develop common 
measures for colleges and universities to assess and report their performance. Approximately 
one-third of states now have in place some form of "performance indicators," with most of 
them developed and implemented within the last three years. In most cases, indicators are 
published in a higher education annual report or "report card," which allows for institutional 
comparisons and provides a readily understandable format for a broad audience. In this 
changing context, the project provided an important opportunity to describe and analyze 
prevailing state practices, and to help shape the emerging policies on higher education 
accountability and institutional improvement. 

BACKGROUND AND ORIGINS AT ECS 

ECS has a long history of involvement with state higher education assessment and 
accountability policy. In 1985 ECS published Transforming the State Role in Undergraduate- 
Education, which, among other things, recommended that (1) a state’s system of higher 
education be monitored to examine the extent to which it meets state goals for undergraduate 
education, and (2) the system report its results periodically. Beginning in 1986, ECS in 
conjunction with the American Association for Higher Education and the State Higher 
Education Executive Officers, undertook periodic national surveys of state assessment 
initiatives. In 1989 the second such survey, part of a FIPSE project to help further the 
national dialogue on higher education assessment, culminated in the publication, Assessing 
College Outcomes - What State Leaders Need to Know, which provided guidance on a 
number of basic questions about using assessment policy to benefit both students and the 
state. 

The "new accountability" project built directly on this past work. ECS’s early interest 
in assessment initiatives reflects the fact that its members and constituencies are drawn from 
both the academic and public policy communities. Founded in 1965, the primary constituents 
of ECS are governors, legislative leaders and their senior policy aides, chief state school 
officers, state higher education executive officers and their senior policy associates, members 
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of state education boards, leaders of local schools, campuses and governing bodies, and others 
appointed by governors as ECS commissioners. (See attachment A for a summary of ECS 
organizational structure, purposes and networks.) 

ECS brought to this project the experience and capacity to: 

• bring a cross-section of relevant stakeholders together to discuss common themes and 
problems across the states; 

• draw on existing research and practices in the area of assessment, and our 
organizational ties to individuals and organizations to carry forward these activities; 
and 

• relate the debate about assessment and accountability to K-12 and other higher 
education concerns as well as to the broader ECS agenda encompassing system 
change. 

Based on ECS’s previous efforts to assist state policymakers in these areas, this 
project sought to identify and analyze the array of performance-based indicators currently in 
use or being developed at the state level. Second, we sought to examine how that array could 
be used by states in determining the effectiveness of institutions at providing a quality 
undergraduate education and in spurring improvements in teaching and learning. Third, we 
sought to synthesize that information into new approaches to help shape more effective state 
policies for undergraduate education that include indicators of student learning outcomes. 

PROJECT DESCRIPTION 

The project involved the following activities, many of which were part of the original 
project planning and others which reflected the changing state policy context. 

1. Establishing a Dialogue on Indicators of Institutional Quality and Effectiveness 

In Chicago on April 5, 1992, ECS convened a technical group comprised of Peter 
Ewell and Dennis Jones (National Center for Higher Education Management Systems 
[NCHEMS]), Theodore Marchese (American Association for Higher Education [AAHE]), 
Zelda Gamson (University of Massachusetts), and Michael Nettles and Marvin Peterson 
(University of Michigan) to discuss the assessment of campus conditions and practices related 
to good undergraduate education. 
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In a discussion moderated by Kay McClenney, vice president of ECS, the group was 
asked to deliberate these questions: What evidence or information would tell you if 
conditions and practices related to good undergraduate education exist on a campus (e.g., how 
could you tell if active learning was occurring or not)? Could this information be translated 
into input, process or outcome indicators that also would be helpful tor institutions to improve 
student learning? The objective was not necessarily to reach a consensus on either the 
conditions or the indicators but rather to get some of the best thinking about this topic on the 
table. 

2. Forming a Project Planning and Advisory Group 

A seven-member planning group was formed to bring together representatives of 
higher education and policymakers to assist in developing a strategic project plan. The group 
included: Sandra Ruppert (project director, ECS), Joni Finney (ECS), Patrick Callan 
(California Higher Education Policy Study Center), Peter Ewell and Dennis Jones, Michael 
Nettles, and Richard C. Richardson Jr. (Arizona State University [ASU|). The planning group 
also met in Chicago on April 6, 1992, to review a proposed work plan and make suggestions 
about specific activities. 

Together, the April 5 and April 6 meetings were very helpful for thinking about how 
to move forward on this project. Results of those meetings were instrumental in developing a 
revised work plan and a comprehensive strategy for addressing project objectives. 

3. Developing a Strategic Work Plan for Assessing System-Level Performance 
The work plan developed in April 1992 included the following components: 

• Developing a conceptual framework for assessing system-level performance as it 
pertains to undergraduate teaching and learning. 

In order to report on the performance of higher education, a conceptual "model" of the 
enterprise and the environment within which it functions was needed as a starting 
point. We envisioned that the conceptual framework(s) would evolve through a series 
of draft models incorporating multiple perspectives on the issue. 

• Testing and refining ideas about system-level performance and its implications for 
state policy by stimulating dialogue among educators and policymakers. 
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Our plans called for engaging in an iterative process to develop a conceptual 
framework for system-level performance. Ideas contained in draft framework(s) would 
set the context for debate among educators and state policymakers in rethinking how 
problems related to accountability and improvement are defined and how alternative 
solutions are envisioned. We planned a series of regional and statewide meetings to 
convene state- and system-level leaders and institutional researchers and administrators 
involved in assessment issues. 

• Assisting state policymakers in advancing an agenda for system-level quality which 
incorporates educational indicators for institutional accountability and improvement. 
The conceptual frameworks and other commissioned papers, were intended to 
encourage states and institutions to engage in "an educational partnership" around a 
shared concept of educational quality. The project would provide state policymakers 
with a set of recommendations for the application and use of performance indicators 
useful for both accountability and improvement purposes. 

4. Developing Draft Conceptual Frameworks 

ECS contracted with Peter Ewell and Dennis Jones (NCHEMS) and Richard C. 
Richardson Jr. (ASU) to draft the conceptual frameworks for discussion. The consultants 
were selected because of their distinct — and complementary — work related to system-level 
quality. Previous work by Ewell/Jones emphasized constructing indicators around the current 
condition of and relationship among (a) students, (b) institutions (and "other providers" of 
postsecondary instruction), and (c) society at large (including employers, agencies, and the 
public generally). The perspective concentrated on indicating the elements of "good practice" 
in undergraduate instruction. 

Richardson’s view of quality was tied to higher education’s ability to accommodate 
the growing diversity that will characterize America’s college-going population in the next 
quarter-century. Access and quality were seen as inseparable. His work emphasized teaching 
and support strategies for promoting comparable achievement among a diverse student body. 

5. Organizing Regional Forums 

Two regional meetings were held to debate and refine the ideas contained in the 
Ewell/Jones and Richardson draft frameworks. The first meeting was June 22, 1992, in 
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Miami Beach, Florida, scheduled to coincide with the AAHE Conference on Assessment in 
Higher Education in order to draw on the expertise of educators and institutional researchers 
attending the conference. 

A second meeting with the same format, but with emphasis on the system or state 
policy perspective, took place on August 5, 1992, in Cincinnati, Ohio, in conjunction with the 
ECS National Forum and Annual Meeting. Invited participants included system- or state-level 
leaders interested in enhancing system-level quality and in using educational indicators for 
measuring performance. 

Elements of the framework and questions addressed at both forums included the 
following: 

• Criteria that define system- or state-level quality — What are the public interests of 
higher education such as access and equity? To what extent are concepts of quality 
shaped by current economic and social conditions and how have they changed over 
time? What are the skills, knowledge and personal attributes that all college graduates 
should hold in common? Are principles of quality management from business and 
industry applicable to higher education systems? 

• Conditions for good undergraduate education — What key characteristics of good 
educational practice, based on educational research, directly relate both to enhanced 
learning and student satisfaction? What are the common themes and areas of 
consensus on conditions and practices? How do state policies influence those 
conditions? 

• Indicators that provide evidence of those conditions — What is the array of 
quantitative and qualitative measures that provide evidence for any designated 
condition of "good practice"? Could certain indicators be meaningfully summarized 
into an "Index of Leading Educational Indicators" that report on the condition of a 
state’s higher education system? How should states use indicators to assess system- 
level quality and to spur institutional improvements in teaching and learning? How 
available or accessible are these data? What is the feasibility for collecting these 
indicators? 
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To facilitate discussion of these issues, ECS commissioned a working document 
intended to provide a "common language" for discussing performance indicators. "Pointing 
the Way: Indicators as Policy Tools in Higher Education" was prepared by Peter Ewell and 
Dennis Jones of NCHEMS. The paper defined indicator systems and described how they 
work, and it provided a set of policy-level questions for reviewing the soundness of any 
particular indicator proposal. The paper was used as a discussion piece for the participants at 
the regional dialogues, and later revised for inclusion in the final project publication. 

6. Analyzing Policy Development Through 10 Case Studies 

Following the regional forums, the planning and advisory group met in Denver to 
discuss the outcomes of the forums and advise ECS stalf on next steps. Attending the 
September 24, 1992, meeting were: Pat Callan, Peter Ewell, Joni Finney (formerly with ECS, 
then with the California Higher Education Policy Center), Dennis Jones, Charles Lenth (then 
with the State Higher Education Executive Officers, currently with ECS), Dewayne Matthews 
(Western Interstate Commission for Higher Education [WICHE]), and Richard Richardson. 

The meeting was facilitated by Sandra Ruppert, ECS/FIPSE project director. 

The meeting was pivotal in terms of reaching several conclusions about the 
development and use of performance indicators. Indicators were already becoming an 
increasingly accepted policy tool, although there was considerable misunderstanding about the 
use of indicators as a "scorecard," and there appeared to be no clear state strategies to develop 
and use indicators to achieve performance improvements. The consensus of the planning 
group, based on recommendations from participants at the forums, was that the project should 
"move beyond dialogue" and examine in depth the public policy environment surrounding the 
design and use of indicators at the state level. 

As a result the work plan was modified to incorporate the development of state-level 
case studies that would interpret prevailing state practices and experiences. The case studies 
were designed to assist in understanding the context within which state policymakers address 
accountability and assessment, while the conceptual frameworks were used to test ideas about 
the state policymaking process. 

In early December 1992, letters were sent to the state higher education executive 
officers in 10 states selected as case-study states. The 10 states (Colorado, Florida, Illinois, 
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Kentucky, New York-SUNY, South Carolina, Tennessee, Texas, Virginia and Wisconsin) 
were selected for the study because some form of qualitative or quantitative performance 
measures were currently used or being developed to assess student learning outcomes and/or 
institutional effectiveness. (See attachment B for copy of letter describing study and list of 
state contacts.) In January, the initial letters were followed up by telephone interviews with 
the state officials or their designees. (See attachment C for interview format and contact list.) 

The written case studies were based on the interviews and materials provided by the 
state agencies. Their purpose was to construct a stale history of the formulation, planning, 
implementation, and review of policies and practices related to the assessment of education 
effectiveness. A format for preparing them was developed in collaboration with project 
consultants, Peter Ewell and Richard Richardson. The case studies were written by Ewell, 
Richardson, Rupperl and Gordon Van de Water (of the Denver-based consulting firm of 
Augenblick, Van de Water and Meyers). 

7. Organizing a Seminar at the ECS National Forum and Annual Meeting 

ECS hosted a seminar on "Indicators of Quality and Accountability lor Higher 
Education Systems" for representatives from the 10 case-study states to discuss current and 
emerging state policies and practices related to the definition, assessment and reporting of 
higher education quality issues. It was held on July 14, 1993, in Pittsburgh in conjunction 
with the ECS National Forum and Annual Meeting. (See attachment D for letter of invitation 
and participant roster.) Four draft case studies (Colorado, Illinois, South Carolina and 
Tennessee) were sent to meeting participants in advance of the seminar in order to get 
participants’ reactions before ECS proceeded with revising them and completing the 
remaining six. Subsequently, drafts of all case studies were sent to the appropriate state 
higher education agency for feedback and review before finalizing the documents. 

8. Publishing the Final Report 

The final report. Charting Higher Education Accountability: A Sourcebook on State- 
Level Performance Indicators , chronicles the experiences of the 10 case-study states. (See 
attachment E for copy of report.) Collectively, the 10 states illustrate both differences and 
similarities in the manner in which each accountability process had originated, how 
performance measures or indicators were developed, and for what purposes the results are 
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being used. The paper by Jones and Ewell, "Pointing the Way: Indicators as Policy Tools in 
Higher Education," opens the report and sets the context for understanding the rise of new 
accountability policies and performance indicators. The revised conceptual frameworks by 
Richardson and by Ewell bring closure to the report by providing analyses and interpretations 
of performance indicator development and use in the 10 states. 

EVALUATION/PROJECT RESULTS 

This project produced visible and demonstrable results from the separate activities 
outlined above and, even more apparent, as a result of the final project publication. Charting 
Higher Education Accountability: A Sourcebook on State-Level Performance Indicators. 

As indicated, the planning and advisory group meetings were central to shaping the 
course this project took over the past 2 1/2 years. The meetings involving state-level 
policymakers, together with the regular input from advisory group members, served as 
mechanisms for interim evaluation and course correction. Although no formal evaluation (or 
summative assessment) of the separate activities was undertaken, periodic feedback served an 
important role in formative assessment and evaluation. 

In preparing the case studies, the state higher education agencies were repeatedly 
contacted to evaluate the accuracy and relevance of our interpretation of developments in their 
state and provide valuable feedback. This activity helped the authors and project director to 
keep the case studies on track and produce a publication that all of the participating states 
would support. 

An even more important indicator of this project’s evaluation and results is the 
widespread use of the materials that were produced. Participants at the meetings often 
reported that they "took home what they learned"; that is, from the discussions about and 
exposure to how assessment/accountability issues were being confronted in other states, they 
were able to shape and redirect some of their own actions. The results from the project 
publication are becoming even more widespread. Copies have been sent to all state higher 
education agencies and other interested parties; in return, numerous statements have been 
received about its value and usefulness and frequent requests for additional copies are made. 
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Several requests for multiple copies also have been received from faculty members to use in 
teaching graduate courses on state policy. 

The findings from this project also have been presented at national and state-level 
meetings. The most recent example was a presentation by Sandra Ruppeit and Charles Lenth 
to the annual Kentucky Governor’s Conference on College Trusteeship, where "quality and 
accountability" were a primary focus for a gathering of more than 200 public college and 
university trustees. Additional presentations and short articles in widely-distributed ECS 
publications (e.g.. State Leader) are planned for the near future. 

Finally, what we and others believe to be the positive evaluation and results of this 
project have convinced ECS to undertake an additional summary report. A 20-page policy 
guide titled. Charting Higher Education Accountability: The Roots and Realities of State- 
Level Performance Indicators , is now in draft form and scheduled lor publication in 
November. This publication, aimed primarily at governors and legislators, legislative and 
executive assistants, state higher education leadership and others in policymaking positions, 
will summarize and set forth the principal findings from the project and the longer case-study 
publication. 

Assessment, institutional accountability and the continuing challenges involved in 
defining and measuring student outcomes continue to be central concerns of ECS and its 
many constituents. The current ECS chairman. Governor Roy Romer of Colorado, frequently 
articulates the need for clear expectations and measurable outcomes in higher education. In 
the coming year, we will be addressing these issues as part of the chairman’s agenda on 
"Quality Counts . . . Setting Expectations for Higher Education and Making Them Count," 
drawing on our past work in assessment and accountability from this project and others. 

SUMMARY AND CONCLUSIONS 

This project has been successful in meeting several important objectives and 
contributing to the continuing development of accountability policies and practices in many 
settings. Specifically: 



• we explored and analyzed the complexities of student assessment and institutional 
accountability policies at the state level, particularly as these have taken shape in the 
development of state higher education performance indicators; 

• we worked with states to relate assessment and accountability to other areas of state 
policy, including academic planning, finance and governance; and 

• we organized numerous meetings, made presentations and produced materials that 
extended the reach of this project to many states, organizations and institutions. 

The interest — particularly by policymakers, elected officials and the public at large 

— in the issues of the quality and performance of colleges and universities is continuing to 
grow. This project provided much of the documentation and analysis of this "accountability 
movement"; the specific issues and concerns have shifted over time while the overall interest 
in performance and accountability has remained a key concern of policymakers. 

To meet the growing demands for performance and accountability, states need to be 
much clearer in articulating their expectations for higher education; and the indicator systems 
need to be more directly tied to these objectives and capable of accurately monitoring 
progress to achieve these results. This challenge will continue for states and for higher 
education institutions, and continued investments are necessary in order to provide long-term 
benefits. ECS will continue to be active and supportive in these areas and will look for 
partners in continuing to achieve progress. 
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Initiatives 

ECS' work with select initiatives 
provides a major vehicle for influencing 
education policy. At any one time, staff 
members are working with people 
throughout the country to address 
major education challenges. This work 
provides an opportunity for ECS to in- 
fluence policy in the states and to help 
education leaders improve their educa- 
tion systems. ECS initiatives include: 

Re:Leaming — A partnership 
among ECS, the Coalition of Essen- 
tial Schools in Providence, Rhode Is- 
land, and participating states, 
districts and schools, to fundamen- 
tally restructure the education sys- 
tem "from the schoolhouse to the 
stateho.use." 

* School Finance — Tracking of 
school finance systems in all 50 
states and research on how educa- 
tion dollars flow from the state level 
to the school site, the implications of 
current budgeting and funding 
processes and the impact of restruc- 
turing of schools on the allocation of 
resources. 

^ Mathematics and Science Improve- 
ment — Collaborative work with na- 
tional leaders in mathematics and 
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issues and trends. The Clearinghouse 
responds to approximately 3,000 in- 
quiries a year from legislators, 
governors' aides, political advisers, 
reporters, national education organiza- 
tions, businesspeople, state and federal 
agencies, school boards and other 
groups, providing them with topical re- 
search and publications for sound 
decision making and reporting. 

A resource library and computerized 
database are maintained on all 50 states 
with information on more than 300 
topics. 

Publications 

A major publisher and distributor of 
education policy materials (both print 
and video), ECS' distribution center 
handles more than 100 titles. Each title 
is distributed free of charge to ECS com- 
missioners and is marketed for sale to 
other constituents, both through direct 
marketing efforts and through other 
publications and forums in cooperation 
with constituent organizations. ECS' 
flagship publication, the State Education 
Leader, is published three times a year, 
with each issue providing an in-depth 
examination of a major education topic. 



Regular and Special 
Meetings 

ECS meetings are one more way that 
we work directly with our constituents 
on issues that affect them. ECS sponsors 
several regular education gatherings: 

■O’ National Forum and Annual Meet- 
ing — Attracts hundreds of educa- 
tion, political and business leaders, 
including ECS commissioners and 
alumni, each year to discuss major is- 
sues affecting K-12 and postsecon- 
dary education. 

■fr Steering Committee Meetings — 
Brings together Steering Committee 
members (one ECS commissioner 
from each state) to discuss impor- 
tant education issues affecting their 
states. 

■fr Special National and Regional Meet- 
ings — Includes meetings such as a 
mathematics and science education 
reform symposium with the Nation- 
al Science Foundation and a school 
finance conference with the National 
Conference of State Legislatures. 

Education Commission of the States 

707 17th Street, Suite 2700 
Denver, Colorado 80202-3427 
303-299-3600 
FAX 303-296-8332 
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The mission of the Education Commis- 
sion of the States (ECS) is to help state 
leaders develop and carry out policies 
that promote improved performance of 



For over 25 years, ECS has been the 
only major education organization that 
does not serve a particular special-inter- 
est group, reaching out to thousands of 
people in literally every state, role 
group and major education organiza- 
tion. This distinction as an organization 
that cuts across roles and levels — local, 
district, state, federal — enables us to 
analyze all sides of an issue and, more . 
important, bring together people with 
diverse perspectives to work with and 
learn from one another. 

A combination of direct person-to-per- 
son contact, program and project work, 
group assistance through meetings and 
public visibility through communica- 
tions work enables ECS to stay on the 
cutting edge of education issues and 
provide continuing assistance to con- 
stituents. It also makes ECS uniquely 
qualified to help policy makers change 
their education systems to serve the 
needs of all children. 



the education system, as reflected in > ,, - 
creased learning by all citizens. 




About ECS 
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Priorities 

ECS is committed to a sustained, multi- 
year action agenda focusing on: 

■> Fundamentally changing the educa- 
tion system to better educate all stu- 
dents for life in the 21st century 

■> Strengthening collaboration by 
identifying and promoting policies 
that support collaboration among 
schools, colleges and other educa- 
tion and social services to serve 
children and youth more effectively 

■> Setting clear, high standards for stu- 
dent learning, the systems that sup- 
port them and the work of ECS 

■> Promoting communication that is 

active, two-way and encourages all 
community members to be involved 
in the dialogue about what is best 
for students 

■> Building an ethic of service, both 
for students to the world around 
them and for ECS to its constituents 

■> Embracing diversity, making 
children our priority and tackling 
the issues of poverty and discrimina- 
tion in our society 




To reach these goals, identify other criti- 
cal education issues and carry out our 
mission of promoting a quality educa- 
tion for all students, ECS conducts 
policy research, surveys and special 
studies; convenes state, regional and na- 
tional forums; maintains an information 
clearinghouse; keeps education issues 
in front of the public through com- 
munications efforts; provides technical 
assistance to states; helps states develop 
policies to improve education; publish- 
es education policy materials; and 
fosters nationwide leadership and col- 
laboration in education. 

Staff 

Headquartered in Denver, Colorado, 
the ECS staff of 55 includes educators, 
policy analysts, economists, testing and 
evaluation experts, government rela- 
tions experts, information and com- 
munication strategy specialists, writers, 
editors, public relations and production 
personnel, research associates and sup- 
port staff. The staff assists ECS' con- 
stituents by gathering, analyzing and 
communicating information about 
varied K-12 and higher education is- 
sues. Campus Compact, an ECS project 
on community service, employs eight 
people and is based at Brown Univer- 
sity in Providence, Rhode Island. 
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unicers (1994-951 

Roy Romer, Chairman; Governor of Colorado 

limTl G ' y h0 T 0n ’ Chairman - Eiect; Governor of Wisconsi, 
J.m Edgar, Immediate Past Chairman; Governor of Illinois 
John Hansen, Vice Chairman; State Senator, Idaho 
jedad F. Robertson, Treasurer; Secretary of Education, Higher 
Education Coordinating Council, Massachusetts 



STAFF. 303^99^600 

President’s Office 

Frank Newman, President, Ext, 624 

Peggy Arkins, Administrative Assistant to the President, Ext. 62 
Maria Bucknam, Secretary, Ext. 628 
Lee Calvert, Administrative Assistant, Ext. 659 
Dee Green, Director of Development/Special Assistant 
to the President, Ext. 61 1 



Vice-President’s Office 

Anna Chan, Administrative Assistant to the Vice President 
Ext. 656 ’ 

Kay McClenney, Vice President, Ext. 605 

Linda Rose, Assistant to the Vice President, Ext. 651 



roncy and Programs Studies 

Sheila Arredondo, Policy Analyst, Ext. 643 

Jane Armstrong, Director of Policy Studies, Assessment/ 
Restructuring, Ext. 632 

Rexford Brown, Senior Fellow, Ext. 652 
Josie Canales, Project Assistant, Ext. 682 

Van Dougherty, Policy Analyst, Systems Changes/Urban 
Initiatives, Ext. 630 

Lyn Hesketh, Project Assistant, Ext. 639 

Chrissy Hussey-Thompson, Project Assistant, Ext 621 

Christine Johnson, Director of Urban Initiative, Ext. 636 

hades Lenth. Director, Higher Education Policy Studies 
bxt. 667 

R E« r, 642 aiCh ’ DireCt ° r ° f P ° licy Studies ’ S y stems Change, 

Sandra Ruppert, Policy Analyst, Higher Education. Ext. 676 
ancy i is. Research Assistant. Higher Education, Ext. 606 
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Re:Learning 

Judy Bray, Systems Coordinator, Ext. 618 
Deborah Clemmons, Systems Coordinator, Ext. 662 
Jennifer Crandall, Project Assistant, Ext. 668 
Ron Rapp, Systems Coordinator, Ext. 670 

State Relations and Clearinghouse 

Sharmi Arora, Library Assistant/Technician, Ext. 615 
Carol Breibart, Administrative Assistant, Ext. 617 
Kathy Christie, Information Clearinghouse Coordinator, Ext. 675 
Mary Fulton, Policy Analyst, Finance, Ext. 679 
Alex Medler, Research Associate, Ext. 635 
Janelle Miller, Research Assistant, Ext. 603 
Chris Pipho, Division Director, Clearinghouse/State 
Relations, Ext. 604 

Gerrit Westervelt, State Relations Director, Ext. 612 J 

Communications 

Arleen Arnsparger, Division Director, Communications, Ext. 653 J 
Elizabeth Holman, Communications Specialist, Ext. 625 | 

Martha Lederer, Project Coordinator, Ext. 647 
Anita Margittai, Communications Assistant, Ext. 627 
Christie McElhinney, Public Relations Manager, Ext. 695 
Sherry Freeland Walker, Publications Director, Ext. 626 
Anna M. West, Production Coordinator, Ext. 655 

Administration 

Stephanie Cardinale, Accountant, Ext. 693 

Barbara Deard, Administrative Assistant/Benefits Administrator, 

Ext. 620 

Judy Feather, Central Services Coordinator, Ext. 692 
Rose Marie Franzen, Network Administrator, Ext. 649 
Antionette Garcia, Accounting Clerk, Ext. 660 
Mary Guidotti, Accountant, Ext 661 
Charley Kahler, Database Programmer, Ext. 663 
Christine Morgenweck, Human Resources Administrator, 

Ext. 666 

Edie Romanski, Receptionist, Ext. 0 

Barbara L. Sharp, Division Director, Administrative Services, 

Ext. 673 

Angela Vidick, Meeting Coordinator, Ext. 609 
Adrienne Welsh, Information Systems Specialist, Ext. 696 
JoAnne Wilkins, Database Administrator, Ext. 633 
Craig Woodruff, Central Services Technician, Ext. 694 

Campus Compact Staff, 401*863*1 1 19 

Sarah Eads, Intern 

Daniel Gardner, Network Director 

Betty Johnson, Office Manager 

Nancy Rhodes, Director 

Melissa Smith, Project Assistant 

Jan Torres, Project Director 

Marie Troppe, Project Associate O A 7/94 
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Chairman 
Hon Evan Bayh 
Governor 
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Chairman Elect 
Hon Jim Edgar 
Governor 
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Vice Chairman 

Hon H Cooper Snyder 

Senator 

Ohio 

Treasurer 
Betty Casior 
Commissioner 
ol Education 
Florida 

President 
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EDUCATION 



COMMISSION 



OFTHESWES 



Dear 9~: 

The Education Commission of the States’ (ECS) initiative on State Level 
Indicators of Undergraduate Educational Effectiveness" is currently 
exploring, with support from the Fund for the Improvement of Postsecondaiy 
Education (FIPSE), some of the critical issues related to educational 
effectiveness and public accountability in undergraduate education. We seek 
your assistance as we undertake a compreb c c-avc review o t how selected 
states have sought and used information about the pcrf o imance of students , 
institutions and systems of higher education to inform policy decisio ns. 

South Carolina and 9 other states (Colorado, Florida, Illinois, Ken; '.y. New 
York-SUNY, Tennessee, Texas, Virginia and Wisconsin) were selected for 
this study because some form of qualitative or quantitative performance 
measures are currently used or being developed to assess student learning 
outcomes and/or institutional effectiveness related to undergraduate teaching 
and learning. For this study, we refer to these measures as indicators of 
educational effectiveness and we believe they may hold some promise for 
addressing both accountability and improvement purposes. We also believe 
that these 10 states will offer some of the best examples of innovative and 
alternative policy approaches that may assist other states in charting a course 
for undergraduate education reform. 

The purposes of this study are to derive from current state experiences with 
accountability, assessment and educational effectiveness the lessons learned 
and the broader implications for improving state policy and practices. ECS 
is assisted in this initiative by Peter Ewell of the National Center on Higher 
Education Management Systems and Richard C. Richardson Jr. of Arizona 
State University. The study is also coordinated with, and is partially funded 
by ECS’s State Policy and College Learning project, supported by the Pew 
Charitable Trusts. 
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Page 2 

December 1 1, 1992 



Our analysis will focus on three major objectives: (1) to update materials collected as part of the 
1989 ECS-SHEEO survey on assessment and accountability and report on how the increasingly 
more difficult choices regarding the allocation of scarce resources are reflected in state policies 
related to accountability, assessment and educational effectiveness; (2) to provide a better 
understanding of how information about educational effectiveness might meet the need for public 
reporting on the conditions of a state’s system of higher education; and (3) to examine how 
information about educational effectiveness can and should contribute to a state’s development of 
policies that collectively create a supportive environment for undergraduate education reform. 

As a first step, we would like to construct a state history of the formulation, planning, 
implementation and review of policies and practices related to the assessment of educational 
effectiveness. We are interested in obtaining materials (such as state master plans, 
accountability or assessment legislation, state level "report cards", etc.) that best reflect a 
state’s experience, particularly since 1989 when the last ECS-SHEEO state survey was 
conducted. We have intentionally left this definition fairly broad since the extent to which this 
information is related to other state policies varies from state to state. 

On the following page is a list of documents available through the ECS-SHEEO Higher Education 
Clearinghouse. We request your assistance in identifying and providing us with the most pertinent 
documents needed to complete the picture. 



ECS also views this effort as a unique opportunity to connect people and information. We hope to 
get your insights and suggestions along the way through forums and papers we develop. We will 
share with you what we are learning by making information available from all the states working 
with us on this study. To start, we will send you a complimentary copy of the bibliography of all 
the state documents we receive as well as any ECS publications on state policy and assessment of 
undergraduate education from the enclosed list. 



We will be calling your office shortly to talk with you or a member of your staff more about this 
study. Please feel free to call me at 303-299-3676 if you have any questions or to request 
publications. I look forward to talking with you soon. 

Sincerely, 



Sandra Ruppert 

Policy Analyst and Project Director 



cc: 



Jim Mingle 
Kay McClenney 
Aims McGuinness 
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ASSESSMENT PUBLICATIONS 
Education Commission of the States 



Assessing College Outcomes: What State Leaders Need to Know. This guide addresses the 
most common questions asked by state policy makers and others about how state assessment 
policy can benefit both college students and the state. It is designed to help state policy ma ers 
understand the questions they should ask about whether, what and how well students are learning. 

1991 (PA-91-3); 24 pp. 

Toward One System of Education: Assessing to Improve, Not Merely Audit by Grant 

Wiggins. The author discusses how assessment can promote a more seamless K- 12/graduate 
school education system. He argues that assessment should improve performance by providing 
feedback and offers 10 propositions for policy makers to consider. 1991 (PA-9 1-2), 38 pp. 

Assessing Progress in Minority Access and Achievement in American Higher Education by 
Michael T. Nettles. The author analyzes education challenges facing minorities and P r 0 P0Ses 
ways to use assessment to expand and enrich their access and learning opportunities. 1991 

(PA-91 -1); 33 pp. 

State Initiatives in Assessment and Outcome Measurement: Tools for reaching and 
Learning in the 1990s. Patterns and profiles of state policy on assessment, instruments and 
approaches, resources, commitments and future plans. 1990 (PA-90- 1); 1 1 1 pp. 

Assessment and Accountability in Higher Education. Summary of conference proceedings 
from the Assessment and Accountability in Higher Education Conference, December 1989. 
(PA-90-2); 28 pp. 

Assessment and the “New Accountability”: A Challenge for Higher Education’s Leadership. 
Discusses changes in the ideas of academic improvement and accountability and the implications 
of those changes for assessment policy. 1990 (PA-90-3); 22 pp. 

State Policy on Assessment: The Linkage to Learning. Explores policy issues relating 

assessment to curriculum, teaching and resource needs. 1990 (PA-90-4); 10 pp. 

State-Based Approaches to Assessment in Undergraduate Education: A Glossary and 

Selected References. Provides state and institutional leaders with a common set of 
definitions for thinking and talking about assessment policy in undergraduate education. 1988 
(PS-88-2); 7 pp. 

State-Based Case Studies of Assessment Initiatives in Undergraduate Education: Chronology 
of Critical Points. Outlines critical junctures in case studies of five states’ approaches to 
assessment in undergraduate education. 1988 (PS-88-3); 11 pp. 

Assessment and Outcomes Measurement — A View From the States. Responses to an ECS 
50-state survey of statewide or systemwide approaches to assessment and outcomes measurement. 
1987 (PS-87- 1); 57 pp. 

O 

ERIC 



33 



EDUCATION COMMISSION OF THE STATES 
ROSTER FOR Indicators Project 
PRINTED ON December II, 1992 



Kenneth H. Ashworth 
Commissioner 

TX Higher Education Coord Board 
P.O. Box 12788, Capitol Station 
Austin, TX 78711 
512-483-6101 
FAX: 512-483-6127 

Gary S. Cox 
Executive Director 
Council on Higher Education 
West Frankfort Office Complex 
1050 U.S. 127 South, #101 
Frankfort, KY 40601 
502-564-3553 
FAX: 502-564-2063 

Gordon K. Davies 
Director 

State Council of Higher Education 
James Monroe Building 
101 North 14th Street 
Richmond, VA 23219 
804-225-2602 
FAX: 804-225-2604 

David A. Longanecker 
Executive Director 
Commission on Higher Education 
1300 Broadway, 2nd Floor 
Denver, CO 80203 
303-866-4034 
FAX: 303-860-9750 



Katharine C. Lyall 
President 

University of Wisconsin System 

1220 Linden Drive 

1700 Van Hise Hall 

Madison, W1 53706 

608-262-2321 

FAX: 608-263-2046 

D. Bruce Johnstone 
Chancellor 

SUNY System Office 
State University Plaza 
Albany, NY 12246 
FAX: 

William B. Proctor 

Executive Director 

Postsecondary Educ Planning Comm 

Florida Education Center 

210 Collins Building 

Tallahassee, FL 32399 

904-488-7894 

FAX: 904-922-5388 

Arliss L. Roaden 
Executive Director 
TN Higher Education Commission 
404 James Robertson Parkway 
Parkway Towers, Suite 1900 
Nashville, TN 37219 
615-741-7562 
FAX: 615-741-6230 
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Fred R. Shehecn 
Commissioner 

Commission on Higher Education 
1333 Main Street, #200 
Columbia, SC 29201 
803-253-6260 
FAX: 803-253-6267 

Richard D. Wagner 

Executive Director 

Board of Higher Education 

4 West Old Capitol Plaza, #500 

Springfield, IL 62701 

217-782-2551 

FAX: 217-782-8548 



